Skip to content

Add Gemini 3.5 Transcribe submission - #57

Merged
soham-sierra merged 3 commits into
sierra-research:mainfrom
jamg7:add-gemini-3.5-transcriber
Sep 18, 2026
Merged

soham-sierra merged 3 commits into
sierra-research:mainfrom
jamg7:add-gemini-3.5-transcriber

Conversation

@jamg7

@jamg7 jamg7 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Submission for Google Gemini 3.5 Transcribe (protocol: batch, region: us, concurrency: 1) across all 5 mu-bench locales:

  • Utterances: 4,270 / 4,270 (en-US: 817, es-MX: 792, tr-TR: 846, vi-VN: 975, zh-CN: 840)
  • Validation: Passed python -m scoring.validate submissions/raw/gemini-3.5-transcriber
  • Provider Script: Added gemini-3.5-transcribe batch provider support in scripts/transcribe.py

@jamg7
jamg7 requested a review from soham-sierra as a code owner September 17, 2026 15:45
@soham-sierra

Copy link
Copy Markdown
Collaborator

/score

@soham-sierra

Copy link
Copy Markdown
Collaborator

/score

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Scoring Results for Gemini-3.5-Transcribe

Locale WER (corpus) Sig. WER
en-US 5.2% 5.5%
es-MX 9.9% 14.4%
tr-TR 8.9% 19.5%
vi-VN 5.6% 10.9%
zh-CN 10.9% 26.1%
Overall 8.1% 15.3%

Results saved to results/gemini-3-5-transcriber/scores.json. A maintainer can re-run scoring with /score.

@soham-sierra
soham-sierra merged commit e199cb6 into sierra-research:main Sep 18, 2026
4 of 5 checks passed

This branch was previously deployed

2 inactive deployments
github-pages — 06bc0ffb Deployed Sep 18, 2026 by soham-sierra via deploy-leaderboard / deploy #19
scoring — 06bc0ffb Deployed Sep 18, 2026 by soham-sierra via score-and-commit #19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants